Skip to content

[PECOBLR-4216] Cache inline Arrow compression metadata - #1660

Draft
prathamesh-db wants to merge 3 commits into
databricks:mainfrom
prathamesh-db:codex/PECOBLR-4216-cache-inline-compression-metadata
Draft

[PECOBLR-4216] Cache inline Arrow compression metadata#1660
prathamesh-db wants to merge 3 commits into
databricks:mainfrom
prathamesh-db:codex/PECOBLR-4216-cache-inline-compression-metadata

Conversation

@prathamesh-db

@prathamesh-db prathamesh-db commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Description

Caches the compression codec from the initial Thrift inline response alongside the Arrow schema, so later result batches can omit optional result-set metadata without failing in compression detection.

If an initial Thrift response omits required metadata, the result factory now rejects it with typed INVALID_STATE name and numeric code before dereferencing it. Inline Arrow components continue to use INLINE_CHUNK_PARSING_ERROR once the Arrow format is known.

Testing

Focused coverage includes 63 tests across the result factory, streaming and lazy inline Arrow paths, response processor, and Arrow utilities. It covers compressed and uncompressed later batches without repeated metadata and verifies the error name and numeric code for missing initial metadata.

The focused suite passed locally before the rebase, and Spotless passed on the rebased branch. CI is validating the rebased head because the local environment could not resolve dependency versions newly added on main.

Telemetry Errors

  • Not applicable — this PR does not add or change a telemetry-visible error.
  • Applicable — the error uses DatabricksDriverErrorCode where appropriate, and any
    new code is uniquely numbered and tested.
  • Applicable — its driver/server/user classification is linked, or maintainer help is
    requested because the author cannot access the classification.

Additional Notes to the Reviewer

Please confirm that INLINE_CHUNK_PARSING_ERROR remains appropriate once the response is known to be inline Arrow, and that INVALID_STATE is appropriate when metadata is absent before the format can be determined. Retry and download behavior are unchanged.

Signed-off-by: Prathamesh Baviskar <prathamesh.baviskar@databricks.com>
Signed-off-by: Prathamesh Baviskar <prathamesh.baviskar@databricks.com>
Signed-off-by: Prathamesh Baviskar <prathamesh.baviskar@databricks.com>
@prathamesh-db
prathamesh-db force-pushed the codex/PECOBLR-4216-cache-inline-compression-metadata branch from 35eb1d8 to d20538a Compare August 26, 2026 13:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant